Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

files cache improvements #8389

Merged

Conversation

ThomasWaldmann
Copy link
Member

@ThomasWaldmann ThomasWaldmann commented Sep 17, 2024

Fixes #8385.

@codecov-commenter
Copy link

codecov-commenter commented Sep 18, 2024

Codecov Report

Attention: Patch coverage is 72.72727% with 33 lines in your changes missing coverage. Please review.

Project coverage is 81.43%. Comparing base (f1a39a0) to head (ec9d412).
Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
src/borg/cache.py 72.81% 22 Missing and 6 partials ⚠️
src/borg/archive.py 55.55% 2 Missing and 2 partials ⚠️
src/borg/archiver/list_cmd.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8389      +/-   ##
==========================================
- Coverage   81.54%   81.43%   -0.12%     
==========================================
  Files          70       70              
  Lines       12717    12733      +16     
  Branches     2297     2310      +13     
==========================================
- Hits        10370    10369       -1     
- Misses       1694     1707      +13     
- Partials      653      657       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann ThomasWaldmann force-pushed the files-cache-from-archive branch 3 times, most recently from 8456761 to 219a47a Compare September 19, 2024 19:39
@ThomasWaldmann ThomasWaldmann marked this pull request as ready for review September 19, 2024 19:46
@ThomasWaldmann ThomasWaldmann changed the title replace files cache by reading previous archive files cache improvements Sep 19, 2024
@ThomasWaldmann ThomasWaldmann force-pushed the files-cache-from-archive branch 2 times, most recently from f1b3c5c to f452b97 Compare September 19, 2024 21:43
- changes to locally stored files cache:

  - store as files.<H(archive_name)>
  - user can manually control suffix via env var
  - if local files cache is not found, build from previous archive.
- enable rebuilding the files cache via loading the previous
  archive's metadata from the repo (better than starting with
  empty files cache and needing to read/chunk/hash all files).
  previous archive == same archive name, latest timestamp in repo.
- remove AdHocCache (not needed any more, slow)
- remove BORG_CACHE_IMPL, we only have one
- remove cache lock (this was blocking parallel backups to same
  repo from same machine/user).

Cache entries now have ctime AND mtime.

Note: TTL and age still needed for discarding removed files.
      But due to the separate files caches per series, the TTL
      was lowered to 2 (from 20).
@ThomasWaldmann ThomasWaldmann force-pushed the files-cache-from-archive branch 2 times, most recently from 341f02d to 28b2a09 Compare September 21, 2024 09:12
if we detect the conditions for this (rare) race,
abort reading the file and retry.

The caller (_process_any) will do up to MAX_RETRIES
before giving up. If it gives up, a warning is logged
and the file is not written to the archive and won't
be memorized in the files cache either.

Thus, the file will be read/chunked/hashed again at
the next borg create run.
we discard all files cache entries referring to files
with timestamps AFTER we started the backup.

so, even in case we would back up an inconsistent file
that has been changed while we backed it up, we would
not have a files cache entry for it and would fully
read/chunk/hash it again in next backup.
@ThomasWaldmann ThomasWaldmann merged commit b862f2b into borgbackup:master Sep 21, 2024
14 of 16 checks passed
@ThomasWaldmann ThomasWaldmann deleted the files-cache-from-archive branch September 21, 2024 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

improve files cache
2 participants